Investigating Active Learning for Short-Answer Scoring

Active learning has been shown to be effective for reducing human labeling effort in supervised learning tasks, and in this work we explore its suitability for automatic short answer assessment on the ASAP corpus. We systematically investigate a wide range of AL settings, varying not only the item selection method but also size and selection of seed set items and batch size. Comparing to a random baseline and a recently-proposed diversitybased baseline which uses cluster centroids as training data, we find that uncertainty-based sampling methods can be beneficial, especially for data sets with particular properties. The performance of AL, however, varies considerably across individual prompts.